Skip to content

fix(elasticsearch): skip bootstrap-only checks when joining existing cluster#149

Merged
Oddly merged 1 commit into
mainfrom
fix/skip-bootstrap-checks-on-join
May 7, 2026
Merged

fix(elasticsearch): skip bootstrap-only checks when joining existing cluster#149
Oddly merged 1 commit into
mainfrom
fix/skip-bootstrap-checks-on-join

Conversation

@Oddly

@Oddly Oddly commented May 7, 2026

Copy link
Copy Markdown
Owner

Summary

Fixes #148. Builds on the same pattern as #146 / #147 (respect elasticsearch_cluster_set_up: true to avoid bootstrap-only behaviour on join-existing-cluster nodes).

Two checks in the role would fail when a fresh node joins an existing cluster:

  1. Odd-master-count check in `tasks/main.yml` — fails when `--limit` covers an even number of master-eligible hosts. The check is meaningful for fresh-bootstrap quorum, irrelevant when joining a cluster that already has its own quorum.
  2. Runtime cluster setup block in `tasks/elasticsearch-security.yml` — tries to authenticate with `elasticsearch_bootstrap_pw` against the joining node's API. On an existing cluster the bootstrap password is no longer the elastic user's password (it has been replaced during the cluster's own initial setup, and replicates via cluster state), so the check fails after 30 retries with `"Bootstrap password authentication failed and this is not the CA host"`.

In both cases the joining node is in fact correctly added to the cluster (verifiable via `_cat/nodes`), so these checks are blocking a healthy outcome.

Change

  • `tasks/main.yml` — added `- not elasticsearch_cluster_set_up | bool` to the `Check count of master nodes` task's `when`. The count is still computed (might be useful elsewhere); only the failing check is suppressed when joining.
  • `tasks/elasticsearch-security.yml` — added `- not elasticsearch_cluster_set_up | bool` to the `Runtime cluster setup (requires running Elasticsearch)` block's `when`. The whole block is bootstrap-only.

Comments added in both places explaining why.

Test plan

  • Manual: with `elasticsearch_cluster_set_up: true` set and `--limit` to a single new master, role completes; node joins the existing cluster and the `_cat/nodes` listing includes it as expected.
  • Manual: same with two new masters in scope (even count) — no longer fails.
  • Existing molecule scenarios should still pass (default `elasticsearch_cluster_set_up` flow is unchanged).

Notes

These three issues (#146, #148) point to a common pattern: the role bundles "first-time security bootstrap" with normal config rendering. Long-term it might be worth splitting them into separate task files, but this minimal fix addresses the immediate blocker.

…cluster

Two checks in the elasticsearch role assume a fresh cluster bootstrap and
fail when a node is added to an existing cluster:

1. Master-count check (tasks/main.yml) requires an odd count of master
   nodes among `ansible_play_hosts_all`. When joining one or two new
   masters at a time, the limited play has 1-2 master-eligible hosts and
   the check fires even though the existing cluster already has its own
   quorum.

2. Runtime cluster setup block (tasks/elasticsearch-security.yml) tries
   the bootstrap password against the ES API, intending to seed elastic
   user, built-in user passwords, etc. On a node joining an existing
   cluster the elastic user already has its real password (replicated
   via cluster state), so bootstrap auth fails and the block aborts
   with: "Bootstrap password authentication failed and this is not the
   CA host".

Both fixes gate the relevant tasks on `elasticsearch_cluster_set_up`,
consistent with the pattern from #146/#147: when the user signals that
the cluster is already bootstrapped, do not run bootstrap-only logic.

Closes #148
@coderabbitai

coderabbitai Bot commented May 7, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@Oddly has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 30 minutes and 11 seconds before requesting another review.

To continue reviewing without waiting, purchase usage credits in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f1b93633-a0bb-4ca8-9fc4-29f093bbac04

📥 Commits

Reviewing files that changed from the base of the PR and between 3148986 and a17870d.

📒 Files selected for processing (2)
  • roles/elasticsearch/tasks/elasticsearch-security.yml
  • roles/elasticsearch/tasks/main.yml
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/skip-bootstrap-checks-on-join

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@Oddly Oddly added the ci:run Trigger gated pull request CI label May 7, 2026
@github-actions github-actions Bot removed the ci:run Trigger gated pull request CI label May 7, 2026
@Oddly Oddly merged commit 39037c0 into main May 7, 2026
15 checks passed
@Oddly Oddly deleted the fix/skip-bootstrap-checks-on-join branch May 7, 2026 17:59
Oddly added a commit that referenced this pull request May 7, 2026
…xisting cluster (#149)" (#150)

This reverts commit 39037c0.

The block-level `when: not elasticsearch_cluster_set_up | bool` on the
"Runtime cluster setup" block in elasticsearch-security.yml regresses
fresh-install scenarios. The block contains an inner set_fact that
flips elasticsearch_cluster_set_up to true partway through, after the
initial passwords file is written. Block-level when is copied to every
inner task and re-evaluated against the current variable state, so
every task after that internal set_fact silently skips — including the
"Fetch elastic password after initial setup" task that sets
elasticstack_password. Downstream consumers (cluster settings,
watermarks, the molecule shared set_ci_watermarks helper) then see
elasticstack_password undefined and either fall back to a hardcoded
file path that doesn't exist on custom-path scenarios, or fail
outright.

Symptom in CI: elasticsearch_custom, elasticsearch_cert_content, and
elasticsearch_diagnostics scenarios fail at the watermark setup step
on rocky9/debian13 + ES 9. Previously green on the same scenarios.

Reverting both files together. The main.yml master-count change is
reverted alongside it because it's only useful in combination with the
join-existing-cluster path that needs further work to land cleanly.
A follow-up PR will gate only the bootstrap-password tasks
individually instead of the whole block.

Reopens #148.
Oddly added a commit that referenced this pull request May 7, 2026
…cluster (#151)

Adds `not elasticsearch_cluster_set_up | bool` to the four bootstrap-password
tasks in elasticsearch-security.yml plus the master-count check in main.yml,
so a joining node skips the checks that only make sense for fresh-bootstrap.
Reopens issue #148 after the previous attempt (#149, reverted in #150)
regressed fresh-install scenarios.

The previous attempt put the gate on the whole "Runtime cluster setup" block.
That block contains an inner `set_fact` that flips elasticsearch_cluster_set_up
to true after the initial passwords file is written. Ansible re-evaluates
block-level whens per task against the live variable, so every inner task
after that flip silently skipped, including the fetch that sets
elasticstack_password. Downstream tasks then saw it undefined and either
fell back to a hardcoded path that doesn't exist on custom-path scenarios,
or failed outright.

Gating per task avoids that — each task's when is independent and doesn't
get re-evaluated against later state. The four gated tasks all already had
`not elasticsearch_passwords_file.stat.exists | bool` in their when, so
they're a clean place for the additional guard.

The master-count check in main.yml was uncontroversial in #149 — it sits
before the security tasks run and the inner flip never reaches it — so it
returns unchanged.

No molecule scenario covers the join-existing-cluster path: it would need
a multi-node setup where the joining node sees the passwords file as absent
(stat delegated to the CA host), which the existing single-node scenarios
can't reproduce. Existing scenarios still validate the fresh-bootstrap path
and would catch a regression like the one #149 introduced.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skip master-count and bootstrap-password checks when joining an existing cluster (elasticsearch_cluster_set_up=true)

1 participant